Artificial Intelligence and Machine Learning

 

COVID Detection using AI

Key Points to Note¶

  • The objective of this week is to experience how businesses use AI with tools & technologies involved to solve problems

  • The outcome of this week is not to understand the tiny details like codes and syntax, but to focus on how an end-to-end AI solution works, the methods, and the final output/results

  • Do not worry about the details of how something was created - you will learn the concepts demonstrated over the duration of this program

  • Think of this week as a preview of what you will be capable of doing at the end of this program

Problem¶

A renowned chain of hospitals is facing a problem of increased overload on resources (human as well as medical) due to a rise in the number of COVID cases. They want your help to build an AI solution that will help them predict the likelihood of a patient being infected by COVID by analyzing a chest X-ray scan image of the patient to identify the patients who are less likely to have COVID and prioritize critical cases.

AI Solution Workflow¶

Tabular Data Case Study - Hotel Booking Cancellation Prediction v1_1 - Google Slides - Google Chrome 30-06-2023 16_08_27.png

Business Context and Problem Definition¶

Business Context¶

First identified in December 2019 in the city of Wuhan in China, COVID is caused by a novel coronavirus known as SARS-CoV-2, after the initial outbreak, COVID rapidly spread within China and subsequently to other countries around the world. The World Health Organization (WHO) declared it a global pandemic in March 2020 due to its widespread transmission and impact on public health. Common symptoms of COVID include fever, cough, fatigue, shortness of breath, muscle or body aches, sore throat, loss of taste or smell, headache, and in severe cases, difficulty breathing or pneumonia. It's important to note that some individuals infected with SARS-CoV-2 may remain asymptomatic, meaning they show no noticeable symptoms but can still spread the virus to others. The severity of COVID can vary from mild to severe. Most individuals experience mild to moderate symptoms and recover without requiring hospitalization. Severe cases may require hospitalization, intensive care, and mechanical ventilation. Testing for COVID involves various methods, including molecular (PCR) and antigen tests, as well as chest X-ray scans.

Problem Statement¶

Traditional testing methods (like the RT-PCR test), despite being the gold-standard, are time-consuming and resource-intensive. The RT-PCR test generally involves 3 stages (sample collection, extraction, PCR) with a 4-hour turnaround time (TAT) for the test results in general. This results in long wait times in detecting COVID and with an increase in the number of screenings, the wait time increases further.

With a rapid increase in the number of cases, hospitals face challenges in terms of staffing and logistics. The healthcare system is grappling with the need for swift and accurate COVID detection to curb the spread of the virus. Testing every case with mild symptoms becomes impractical and it becomes important to set up criteria for RT-PCR testing to reduce the turnaround time of test results and control the spread of the virus.

A renowned chain of hospitals have decided to leverage AI to deal with the crisis situation. They want to build an AI system to predict the chances of a patient being COVID positive by identifying patterns in chest X-ray scans.

Solution Approach¶

The AI solution approach involves the scanning of chest X-rays of patients to detect patterns commonly found in COVID-infected people. Conducting a chest X-ray scan and obtaining the resultant scan image for a patient requires approximately 15 minutes. The chest X-rays can be passed to our AI solution and based on the likelihood of COVID.

This methodology can provide the following advantages to hospital:

  • Segregating cases of mild symptoms with lower chances of COVID from the critical cases
  • Reduced TAT for test results, leading to quicker diagnosis
  • Early detection of signs of COVID infection, even before symptoms become severe
  • Resource-efficiency as X-ray machines are commonly available in medical facilities
  • Isolation of patients to contain the further spread of COVID to unaffected people

The X-ray based detection can be used in conjunction with RT-PCR testing to increase the accuracy of diagnosis.

Data Gathering¶

s2.PNG

image.png

The data needed for buidling any AI solution is usually obtained from multiple sources.

In the current scenario, we have the following sources:

  • Medical Imaging Data: This includes infomration such as X-rays, CT scans, MRI scans, ultrasounds, and other types of medical images used for diagnosing and monitoring conditions.

  • Patient Health Records: This includes information such as patient medical history, medications, and test results.

  • Hospital Administrative Records: This includes data related to scheduling appointments, managing patient admissions and discharges, billing, insurance claims, and other administrative tasks.

  • Hospital Medical Resource Records: Includes information about medical supplies, equipment inventory, maintenance schedules, and usage. These records optimize resource allocation, enhance patient care, streamline operations, and ensure timely availability of critical medical assets while adhering to efficient inventory management practices.

  • Clinical Laboratory Test Data: Data generated from various medical tests, such as blood tests, urine tests, tissue samples, and genetic tests.

  • Patient Insurance Records: This includes data related to patients' health insurance coverage, claims, and reimbursement.

The data from different sources are collected and stored in an organized and secure manner in databases and PACS (Picture Archiving and Communication Systems) repositories. Once the data is stored, we can extract necessary data in multiple ways.

  • Data from Databases
    • Export as CSV/Excel File: This method allows for exporting a selected subset or the entire dataset in a CSV (Comma-Separated Values) or Excel file format, which can be easily opened and analyzed using spreadsheet software or using programming languages like Python.
    • Querying from the database: This involves running SQL (Structured Query Language) queries on the database to retrieve specific data based on predefined conditions, allowing for more targeted and customized data extraction for analysis or reporting purposes. The SQL queries can be executed using programming languages like Python by establishing a connection to the database
  • Data from PACS
    • Extracting data from a Picture Archiving and Communication System (PACS) database involves several steps. Ensure you have access authorization and are familiar with DICOM (Digital Imaging and Communications in Medicine) protocols. Identify relevant DICOM tags and use the DICOM Query/Retrieve protocol to query for specific images. Retrieve the images, potentially transferring them locally. Utilize DICOM toolkits in programming languages like Python to construct queries, receive responses, and process images.

For the scope of this session, we will be using X-ray images extracted from a PACS to showcase the end-to-end solution.¶

  • The X-ray images were sampled randomly from the PACS
  • We'll load the data sample into a Python notebook for illustration.

Importing the necessary libraries¶

In [6]:
# installing the library for deploying AI models
!pip install gradio -q
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50.4/50.4 kB 4.0 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.1/18.1 MB 105.5 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 318.7/318.7 kB 27.9 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 94.6/94.6 kB 7.9 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 76.4/76.4 kB 7.6 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.9/77.9 kB 7.3 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 141.9/141.9 kB 13.3 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.9/10.9 MB 112.2 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 kB 5.7 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.3/58.3 kB 5.3 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 71.5/71.5 kB 6.3 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 130.2/130.2 kB 10.1 MB/s eta 0:00:00
In [7]:
# this is necessary for training and building machine learning models
!pip install --upgrade tensorflow
Requirement already satisfied: tensorflow in /usr/local/lib/python3.10/dist-packages (2.17.0)
Requirement already satisfied: absl-py>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.4.0)
Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.6.3)
Requirement already satisfied: flatbuffers>=24.3.25 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (24.3.25)
Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (0.6.0)
Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (0.2.0)
Requirement already satisfied: h5py>=3.10.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (3.11.0)
Requirement already satisfied: libclang>=13.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (18.1.1)
Requirement already satisfied: ml-dtypes<0.5.0,>=0.3.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (0.4.1)
Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (3.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from tensorflow) (24.1)
Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (3.20.3)
Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (2.32.3)
Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from tensorflow) (71.0.4)
Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.16.0)
Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (2.4.0)
Requirement already satisfied: typing-extensions>=3.6.6 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (4.12.2)
Requirement already satisfied: wrapt>=1.11.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.16.0)
Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.64.1)
Requirement already satisfied: tensorboard<2.18,>=2.17 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (2.17.0)
Requirement already satisfied: keras>=3.2.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (3.4.1)
Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (0.37.1)
Requirement already satisfied: numpy<2.0.0,>=1.23.5 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.26.4)
Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from astunparse>=1.6.0->tensorflow) (0.44.0)
Requirement already satisfied: rich in /usr/local/lib/python3.10/dist-packages (from keras>=3.2.0->tensorflow) (13.8.1)
Requirement already satisfied: namex in /usr/local/lib/python3.10/dist-packages (from keras>=3.2.0->tensorflow) (0.0.8)
Requirement already satisfied: optree in /usr/local/lib/python3.10/dist-packages (from keras>=3.2.0->tensorflow) (0.12.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorflow) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorflow) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorflow) (2.2.3)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorflow) (2024.8.30)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.18,>=2.17->tensorflow) (3.7)
Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.18,>=2.17->tensorflow) (0.7.2)
Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.18,>=2.17->tensorflow) (3.0.4)
Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.10/dist-packages (from werkzeug>=1.0.1->tensorboard<2.18,>=2.17->tensorflow) (2.1.5)
Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich->keras>=3.2.0->tensorflow) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich->keras>=3.2.0->tensorflow) (2.18.0)
Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich->keras>=3.2.0->tensorflow) (0.1.2)
In [8]:
# Importing libraries
from IPython import display                                             # Enhanced interactive Python shell
import numpy as np                                                      # Numerical operations with arrays and matrices
import pandas as pd                                                     # Data manipulation and analysis library
import tensorflow as tf                                                 # Framework that provides tools for building machine learning models.
import os                                                               # Interact with file systems
import math                                                             # Mathematical functions and utilities

# Exploratory data analysis
import matplotlib.pyplot as plt                                         # Data visualization library
import seaborn as sns                                                   # Statistical graphics
from tensorflow.keras.preprocessing.image import ImageDataGenerator     # Image data generator with data augmentation
import cv2                                                              # Image processing library
from PIL import ImageFilter                                             # Image filters for image manipulation

# Model building
from tensorflow.keras.layers import Normalization, TextVectorization, StringLookup  # Specific preprocessing layers
from sklearn.metrics import recall_score, confusion_matrix              # Libraries for model evaluation

# Model deployment
import joblib                                                           # Save and load machine learning models
import gradio as gr                                                     # UI for model deployment

# To generate reproducible results
import random as python_random                                          # Generate random data
np.random.seed(42)                                                      # Set seed for NumPy random generator
tf.random.set_seed(42)                                                  # Set seed for TensorFlow random generator

Loading the data¶

In [10]:
# Importing using Google Colab

# Connecting Google Drive to this Python notebook
from google.colab import drive
drive.mount('/content/drive')
Mounted at /content/drive
In [11]:
# defining the dimensions of the images
height, width = 224, 224
# defining how many images to load in one go
batch_size=64

def generate_data(DIR):
    # ImageDataGenerator is a function from tensorflow to perform rescaling, resizing, rotation, and other transformations.
    datagen = ImageDataGenerator(rescale=1./255.)

    # flow from directory is used to generate batches of image data from a directory structure, allowing for efficient loading and preprocessing of image data during model training or evaluation.
    generator = datagen.flow_from_directory(
        DIR,
        batch_size=batch_size,
        shuffle=True,                                                          # shuffles the order of images while generation
        seed=42,
        class_mode='sparse',
        target_size=(height, width),
        classes={'Normal': 0, 'Viral Pneumonia': 1,'Covid': 2}
    )
    return generator

# defining the file path where training and testing images are stored
TRAINING_DIR = '/content/drive/My Drive/Texas McCombs/Python Foundations/Case Studies/CaseStudy - COVID Detection/X-ray Data/train'
TESTING_DIR = '/content/drive/My Drive/Texas McCombs/Python Foundations/Case Studies/CaseStudy - COVID Detection/X-ray Data/test'

train_data = generate_data(TRAINING_DIR)                                       # generate data for training
test_data = generate_data(TESTING_DIR)                                         # generate data for testing

total_image = np.concatenate([train_data.labels,test_data.labels])             # used to access the labels associated with the images
Found 231 images belonging to 3 classes.
Found 66 images belonging to 3 classes.

Exploratory Data Analysis¶

s3.PNG

Exploratory Data Analysis (EDA) plays a very important role in an end-to-end AI solution. It enables

  • Understanding the Data
  • Identifying Data Patterns and Insights
  • Feature Selection and Engineering

EDA¶

Distrbution of Image Types¶

In [12]:
counts = {
    'Normal': len(np.where(total_image == 0)[0]),                              # Calculating the count of Normal images
    'Viral Pneumonia': len(np.where(total_image == 1)[0]),                     # Calculating the count of Viral Pneumonia images
    'COVID': len(np.where(total_image == 2)[0])                                # Calculating the count of COVID images
}

# Extract class labels and corresponding counts
class_labels = list(counts.keys())
class_counts = list(counts.values())

# Create the bar plot
plt.bar(class_labels, class_counts)
plt.xlabel('Classes')
plt.ylabel('Counts')
plt.title('Distribution of Image Types');

Visual Inspection of Images¶

In [13]:
def image_plot(generator, images_per_class):
    # Get the mapping of class names to indices
    class_indices = generator.class_indices
    # Extract the class names from the indices
    class_names = list(class_indices.keys())

    # Create a figure to display the images
    plt.figure(figsize=(15, 10))

    # Loop through each class index and its corresponding class name
    for class_idx, class_name in enumerate(class_names):
        # Get the indices of images belonging to the current class
        class_indices = [i for i, value in enumerate(generator.classes) if value == class_idx]
        # Select a subset of indices for the current class based on 'images_per_class'
        selected_indices = class_indices[:images_per_class]

        # Loop through the selected indices and display the images
        for i, idx in enumerate(selected_indices):
            # Create a subplot for each image
            ax = plt.subplot(len(class_names), images_per_class, class_idx * images_per_class + i + 1)
            # Load and display the image
            img = plt.imread(generator.filepaths[idx])
            plt.imshow(img)
            plt.title(class_name)
            plt.axis("off")

# Assuming 'train_data' is your image generator
images_per_class = 2
image_plot(train_data, images_per_class)
plt.show()

EDA Results¶

Distribution of Image Types

image.png

Types of Images

image.png

Identifying COVID vs non-COVID cases through manual observation of medical images poses a formidable challenge due to the subtle visual differences. The human eye's limitations can lead to extended diagnosis times and potential inaccuracies, especially when dealing with a considerable volume of patient data. Additionally, this process can be time-consuming and prone to subjectivity.

So, we need an AI model that can do the following:

  • Take the X-ray image as input
  • Learn the patterns from the input images
  • Build a mathematical model using these patterns to forecast the likelihood of a patient having COVID
  • Predicting the probability of COVID infection for new cases

image.png

Model Building¶

model building.PNG

The AI model is the 'heart' of our AI solution. The model serves as the core component that brings intelligence and functionality to an end-to-end AI solution. It leverages learned patterns and insights to generate predictions or perform tasks, enabling organizations to make data-driven decisions, automate processes, and unlock valuable insights from their data.

The model building step of an AI solution can be further broken down into the sub-steps shown below.

model building lifecycle.png

Model Training and Evalution¶

Training an AI model is important because it allows machines to learn and perform tasks without explicit programming. It enables the following:

  • Learning from Data
  • Generalization and Adaptability
  • Optimization and Performance Improvement

Model Training and Evaluation¶

In [ ]:
# defining the model
tf.keras.backend.clear_session()                                               # Clears the Keras session to remove any existing models or layers

input_shape = (height, width, 3)                                               # Defines the input shape for the model (height, width, number of channels)
base_model = tf.keras.applications.vgg16.VGG16(
    weights='imagenet',                                                        # Loads the pre-trained weights of the VGG16 model trained on ImageNet dataset
    include_top=False,                                                         # Excludes the top (fully connected) layers of the VGG16 model
    input_shape=input_shape
)
base_model.trainable = False                                                   # Freezes the weights of the VGG16 model to prevent further training

ai_model = tf.keras.Sequential()                                            # Creates a sequential model
ai_model.add(base_model)                                                    # Adds the VGG16 base model to the sequential model
ai_model.add(tf.keras.layers.GlobalAveragePooling2D())                      # Adds a global average pooling layer

ai_model.add(tf.keras.layers.Flatten())                                     # Flattens the input for the subsequent fully connected
ai_model.add(tf.keras.layers.Dense(3, activation='softmax'))                # Adds the final fully connected layer with 3 units and softmax activation

ai_model.compile(loss='SparseCategoricalCrossentropy',                      # Specifies the loss function for trainin
              optimizer=tf.keras.optimizers.Adam(0.001),                       # Sets the optimizer (Adam) and learning rate
              metrics=['acc'])                                                 # Specifies the metrics to evaluate the model's performance
ai_model.summary()

Model Training

In [ ]:
import random
seed_value = 42
random.seed(seed_value)
np.random.seed(seed_value)
tf.random.set_seed(seed_value)
checkpoint = tf.keras.callbacks.ModelCheckpoint('model/ai_model_best.saved', monitor='acc', verbose=1, mode='max',save_best_only=True) # defines file path to save the best model weights, define accuracy metric, verbose display updates of the model and saves best model
early = tf.keras.callbacks.EarlyStopping(monitor="val_loss", mode="min",restore_best_weights=True, patience=5)  #stops the model training if there is no improvement in metrics

callbacks_list = [checkpoint,early]

history = ai_model.fit(                                                     # Fit the model on training data
    train_data,                                                            # Training data generator
    validation_data = test_data,                                           # validation data generator
    epochs=25,                                                             # no of epochs to train the model
    shuffle=False,
    verbose=True,                                                          # Prints updates during training.
    callbacks=callbacks_list
)
Epoch 1/25
4/4 [==============================] - ETA: 0s - loss: 1.4845 - acc: 0.2597
Epoch 1: acc improved from -inf to 0.25974, saving model to model/ai_model_best.saved
4/4 [==============================] - 31s 4s/step - loss: 1.4845 - acc: 0.2597 - val_loss: 1.3230 - val_acc: 0.3030
Epoch 2/25
4/4 [==============================] - ETA: 0s - loss: 1.2610 - acc: 0.2597
Epoch 2: acc did not improve from 0.25974
4/4 [==============================] - 7s 2s/step - loss: 1.2610 - acc: 0.2597 - val_loss: 1.1859 - val_acc: 0.3030
Epoch 3/25
4/4 [==============================] - ETA: 0s - loss: 1.1253 - acc: 0.2771
Epoch 3: acc improved from 0.25974 to 0.27706, saving model to model/ai_model_best.saved
4/4 [==============================] - 7s 2s/step - loss: 1.1253 - acc: 0.2771 - val_loss: 1.1197 - val_acc: 0.2879
Epoch 4/25
4/4 [==============================] - ETA: 0s - loss: 1.0578 - acc: 0.4545
Epoch 4: acc improved from 0.27706 to 0.45455, saving model to model/ai_model_best.saved
4/4 [==============================] - 9s 2s/step - loss: 1.0578 - acc: 0.4545 - val_loss: 1.0973 - val_acc: 0.3939
Epoch 5/25
4/4 [==============================] - ETA: 0s - loss: 1.0306 - acc: 0.4805
Epoch 5: acc improved from 0.45455 to 0.48052, saving model to model/ai_model_best.saved
4/4 [==============================] - 7s 2s/step - loss: 1.0306 - acc: 0.4805 - val_loss: 1.0912 - val_acc: 0.3939
Epoch 6/25
4/4 [==============================] - ETA: 0s - loss: 1.0126 - acc: 0.4805
Epoch 6: acc did not improve from 0.48052
4/4 [==============================] - 7s 2s/step - loss: 1.0126 - acc: 0.4805 - val_loss: 1.0833 - val_acc: 0.3939
Epoch 7/25
4/4 [==============================] - ETA: 0s - loss: 0.9999 - acc: 0.4805
Epoch 7: acc did not improve from 0.48052
4/4 [==============================] - 6s 1s/step - loss: 0.9999 - acc: 0.4805 - val_loss: 1.0691 - val_acc: 0.3939
Epoch 8/25
4/4 [==============================] - ETA: 0s - loss: 0.9833 - acc: 0.4805
Epoch 8: acc did not improve from 0.48052
4/4 [==============================] - 5s 1s/step - loss: 0.9833 - acc: 0.4805 - val_loss: 1.0494 - val_acc: 0.3939
Epoch 9/25
4/4 [==============================] - ETA: 0s - loss: 0.9635 - acc: 0.4805
Epoch 9: acc did not improve from 0.48052
4/4 [==============================] - 6s 1s/step - loss: 0.9635 - acc: 0.4805 - val_loss: 1.0256 - val_acc: 0.3939
Epoch 10/25
4/4 [==============================] - ETA: 0s - loss: 0.9431 - acc: 0.4805
Epoch 10: acc did not improve from 0.48052
4/4 [==============================] - 9s 3s/step - loss: 0.9431 - acc: 0.4805 - val_loss: 1.0012 - val_acc: 0.3939
Epoch 11/25
4/4 [==============================] - ETA: 0s - loss: 0.9239 - acc: 0.4848
Epoch 11: acc improved from 0.48052 to 0.48485, saving model to model/ai_model_best.saved
4/4 [==============================] - 7s 2s/step - loss: 0.9239 - acc: 0.4848 - val_loss: 0.9797 - val_acc: 0.3939
Epoch 12/25
4/4 [==============================] - ETA: 0s - loss: 0.9061 - acc: 0.5065
Epoch 12: acc improved from 0.48485 to 0.50649, saving model to model/ai_model_best.saved
4/4 [==============================] - 7s 2s/step - loss: 0.9061 - acc: 0.5065 - val_loss: 0.9617 - val_acc: 0.3939
Epoch 13/25
4/4 [==============================] - ETA: 0s - loss: 0.8895 - acc: 0.5628
Epoch 13: acc improved from 0.50649 to 0.56277, saving model to model/ai_model_best.saved
4/4 [==============================] - 9s 2s/step - loss: 0.8895 - acc: 0.5628 - val_loss: 0.9452 - val_acc: 0.4394
Epoch 14/25
4/4 [==============================] - ETA: 0s - loss: 0.8749 - acc: 0.6277
Epoch 14: acc improved from 0.56277 to 0.62771, saving model to model/ai_model_best.saved
4/4 [==============================] - 7s 2s/step - loss: 0.8749 - acc: 0.6277 - val_loss: 0.9301 - val_acc: 0.4697
Epoch 15/25
4/4 [==============================] - ETA: 0s - loss: 0.8599 - acc: 0.6537
Epoch 15: acc improved from 0.62771 to 0.65368, saving model to model/ai_model_best.saved
4/4 [==============================] - 9s 2s/step - loss: 0.8599 - acc: 0.6537 - val_loss: 0.9181 - val_acc: 0.4697
Epoch 16/25
4/4 [==============================] - ETA: 0s - loss: 0.8452 - acc: 0.6537
Epoch 16: acc did not improve from 0.65368
4/4 [==============================] - 5s 1s/step - loss: 0.8452 - acc: 0.6537 - val_loss: 0.9049 - val_acc: 0.4848
Epoch 17/25
4/4 [==============================] - ETA: 0s - loss: 0.8301 - acc: 0.6623
Epoch 17: acc improved from 0.65368 to 0.66234, saving model to model/ai_model_best.saved
4/4 [==============================] - 8s 2s/step - loss: 0.8301 - acc: 0.6623 - val_loss: 0.8942 - val_acc: 0.5152
Epoch 18/25
4/4 [==============================] - ETA: 0s - loss: 0.8172 - acc: 0.6623
Epoch 18: acc did not improve from 0.66234
4/4 [==============================] - 6s 1s/step - loss: 0.8172 - acc: 0.6623 - val_loss: 0.8835 - val_acc: 0.5303
Epoch 19/25
4/4 [==============================] - ETA: 0s - loss: 0.8038 - acc: 0.6623
Epoch 19: acc did not improve from 0.66234
4/4 [==============================] - 6s 1s/step - loss: 0.8038 - acc: 0.6623 - val_loss: 0.8698 - val_acc: 0.6061
Epoch 20/25
4/4 [==============================] - ETA: 0s - loss: 0.7906 - acc: 0.7013
Epoch 20: acc improved from 0.66234 to 0.70130, saving model to model/ai_model_best.saved
4/4 [==============================] - 7s 2s/step - loss: 0.7906 - acc: 0.7013 - val_loss: 0.8569 - val_acc: 0.6515
Epoch 21/25
4/4 [==============================] - ETA: 0s - loss: 0.7779 - acc: 0.7273
Epoch 21: acc improved from 0.70130 to 0.72727, saving model to model/ai_model_best.saved
4/4 [==============================] - 7s 2s/step - loss: 0.7779 - acc: 0.7273 - val_loss: 0.8445 - val_acc: 0.6667
Epoch 22/25
4/4 [==============================] - ETA: 0s - loss: 0.7658 - acc: 0.7532
Epoch 22: acc improved from 0.72727 to 0.75325, saving model to model/ai_model_best.saved
4/4 [==============================] - 8s 2s/step - loss: 0.7658 - acc: 0.7532 - val_loss: 0.8325 - val_acc: 0.6818
Epoch 23/25
4/4 [==============================] - ETA: 0s - loss: 0.7541 - acc: 0.7879
Epoch 23: acc improved from 0.75325 to 0.78788, saving model to model/ai_model_best.saved
4/4 [==============================] - 7s 2s/step - loss: 0.7541 - acc: 0.7879 - val_loss: 0.8205 - val_acc: 0.7121
Epoch 24/25
4/4 [==============================] - ETA: 0s - loss: 0.7438 - acc: 0.8398
Epoch 24: acc improved from 0.78788 to 0.83983, saving model to model/ai_model_best.saved
4/4 [==============================] - 9s 2s/step - loss: 0.7438 - acc: 0.8398 - val_loss: 0.8085 - val_acc: 0.7727
Epoch 25/25
4/4 [==============================] - ETA: 0s - loss: 0.7324 - acc: 0.8442
Epoch 25: acc improved from 0.83983 to 0.84416, saving model to model/ai_model_best.saved
4/4 [==============================] - 7s 2s/step - loss: 0.7324 - acc: 0.8442 - val_loss: 0.7989 - val_acc: 0.7727

Model Evaluation

In [ ]:
#model accuracy
ytrain = np.array([])                                                          # Initialize an empty array for storing true labels of test data
xtrain = []                                                                    # Initialize an empty list for storing test data

for i in range(math.ceil(len(train_data.classes)/batch_size)):                 # Loop over test generator batches to extract test data and true labels
    xtrain.append(train_data[i][0])                                            # Append test data to xtest list
    ytrain= np.concatenate((ytrain,train_data[i][-1]))                         # Concatenate true labels to ytest array

xtrain = np.concatenate((xtrain),axis=0)                                       # Concatenate test data along the batch axis

ypred_prob_train =ai_model.predict(xtrain)                                  # Predict probabilities for the test data
ypred_train = np.argmax(ypred_prob_train,axis=1)                               # Predicted labels by selecting the class with the highest probability
8/8 [==============================] - 9s 405ms/step
In [ ]:
model_train_score = recall_score(ytrain, ypred_train, average='macro')
print("Model Score on Train Data:", np.round(100*model_train_score, 2))
Model Score on Train Data: 81.88
In [ ]:
# Set the size of the figure for the heatmap
plt.figure(figsize=(6, 6))
# Compute the confusion matrix based on true and predicted labels
hm = sns.heatmap(confusion_matrix(ytrain,ypred_train), annot=True, vmin=0, fmt='g', cmap='Blues', cbar=False,xticklabels=['Normal','Viral Penumonia','Covid'],yticklabels=['Normal','Viral Penumonia','Covid'])
hm.set(xlabel='Predicted labels') # Set the x-axis label for the heatmap
hm.set(ylabel='True label') # # Set the y-axis label for the heatmap
Out[ ]:
[Text(45.72222222222221, 0.5, 'True label')]
In [ ]:
#model accuracy
ytest = np.array([])                                                           # Initialize an empty array for storing true labels of test data
xtest = []                                                                     # Initialize an empty list for storing test data

for i in range(math.ceil(len(test_data.classes)/batch_size)):                  # Loop over test generator batches to extract test data and true labels
    xtest.append(test_data[i][0])                                              # Append test data to xtest list
    ytest= np.concatenate((ytest,test_data[i][-1]))                            # Concatenate true labels to ytest array

xtest = np.concatenate((xtest),axis=0)                                         # Concatenate test data along the batch axis

ypred_prob_test =ai_model.predict(xtest)                                    # Predict probabilities for the test data
ypred_test = np.argmax(ypred_prob_test,axis=1)                                 # Predicted labels by selecting the class with the highest probability
3/3 [==============================] - 0s 109ms/step
In [ ]:
model_test_score = recall_score(ytest, ypred_test, average='macro')
print("Model Score on test Data:", np.round(100*model_train_score, 2))
Model Score on test Data: 81.88
In [ ]:
# Set the size of the figure for the heatmap
plt.figure(figsize=(6, 6))
# Compute the confusion matrix based on true and predicted labels
hm = sns.heatmap(confusion_matrix(ytest, ypred_test), annot=True, vmin=0, fmt='g', cmap='Blues', cbar=False,xticklabels=['Normal','Viral Penumonia','Covid'],yticklabels=['Normal','Viral Penumonia','Covid'])
hm.set(xlabel='Predicted labels') # Set the x-axis label for the heatmap
hm.set(ylabel='True label') # # Set the y-axis label for the heatmap
Out[ ]:
[Text(45.72222222222221, 0.5, 'True label')]

Understanding Model Predictions and Business Consequences¶

image.png

image.png

It is much more important to ensure that patients who actually have the infection are correctly identified based on their X-ray scan, even if it results in a few false COVID detections

  • The stakes are higher if the model identifies someone actually having COVID as non-COVID when compared to the case of people not having the infection being tagged as COVID positive

Observations from Model Evaluation¶

Our AI model has a score of 84% on the train data and 75% on the test data respectively.

  • This score is pretty low considering our current scenario

  • This becomes a worry as the ultimate goal would be to make predictions for new patients, and we do not want a model that will fail to predict COVID cases then.

Model Tuning¶

Model tuning is important for

  • Optimizing Performance
  • Determining the right fit
  • Adapting to Data Characteristics

In the current scenario, we are going to undertake two steps to tune the model

  1. Since the number of images to train the data with is low, we will be 'synthetically creating' new data by applying various transformations to the existing images. These transformations create new variations of the data while maintaining the original label, resulting in a more diverse and robust training dataset. This helps in

    • Increased Generalization: By exposing it to different variations of the same data, the model generalizes better to new, unseen data. This helps prevent the scenario where the model becomes too specialized to the training data and performs poorly on new data.

    • Improved Robustness: By introducing variations in the training data, models become more robust to different lighting conditions, orientations, scales, and other factors that might be encountered in real-world scenarios.

image.png

  1. We'll update the base model by making additions to the architecture of the model.

Model Tuning¶

In [ ]:
# function to visualize the images
def image_plot(generator, image_numbers):
    img_feature = generator[0][0][:image_numbers]
    img_label = generator[0][1][:image_numbers]

    plt.figure(figsize=(20, 15))
    for i in range(image_numbers):
        ax = plt.subplot(2, 3, i + 1)  # Use 2 rows and 3 columns for the subplot grid
        plt.imshow(img_feature[i])
        plt.title("Normal" if img_label[i] == 0 else "Viral Pneumonia" if img_label[i] == 1 else "Covid")
        plt.axis("off")
In [ ]:
# adding image augmentation
def generate_data_augmented(DIR):
    datagen = ImageDataGenerator(                                              # Image data generator is a function from tensorflow to perform rescaling, resizing, rotation, and other transformations.
        rescale=1./255.,                                                       # Rescale pixel values to [0, 1]
        zoom_range=0.2,                                                        # Randomly zooms images by a factor of 0.1
        rotation_range=25,                                                     # Randomly rotates images by up to 20 degrees
        width_shift_range=0.15,                                                # Randomly shifts images horizontally by 10% of the total width
        height_shift_range=0.15,                                               # Randomly shifts images vertically by 10% of the total height
        horizontal_flip = True                                                 # Randomly flips images horizontally
    )
    generator = datagen.flow_from_directory(
        TRAINING_DIR,                                                          # Directory containing the training images
        batch_size=batch_size,                                                 # Batch size for generating augmented data
        seed=42,                                                               # Seed value for random number generation
        class_mode='binary',                                                   # Type of class assignment ('binary' in this case)
        target_size=(height, width),                                           # Desired size (height, width) for the images
        classes={'Normal': 0, 'Viral Pneumonia': 1,'Covid': 2}                 # Mapping of class names to numerical labels
    )
    return generator

aug_train_data = generate_data_augmented(TRAINING_DIR)

image_plot(aug_train_data,6)
Found 231 images belonging to 3 classes.
In [ ]:
# defining the model architecture
tf.keras.backend.clear_session()                                               # Clears the Keras session to remove any existing models or layers

input_shape = (height, width, 3)                                               # Defines the input shape for the model (height, width, number of channels)
base_model = tf.keras.applications.vgg16.VGG16(
    weights='imagenet',                                                        # Loads the pre-trained weights of the VGG16 model trained on ImageNet dataset
    include_top=False,                                                         # Excludes the top (fully connected) layers of the VGG16 model
    input_shape=input_shape
)
base_model.trainable = False                                                   # Freezes the weights of the VGG16 model to prevent further training

tuned_ai_model = tf.keras.Sequential()                                            # Creates a sequential model
tuned_ai_model.add(base_model)                                                    # Adds the VGG16 base model to the sequential model
tuned_ai_model.add(tf.keras.layers.GlobalAveragePooling2D())                      # Adds a global average pooling layer

tuned_ai_model.add(tf.keras.layers.Flatten())                                     # Flattens the input for the subsequent fully connected
tuned_ai_model.add(tf.keras.layers.Dense(256, activation='relu'))                 # Adds a fully connected layer with 256 units and ReLU activation
tuned_ai_model.add(tf.keras.layers.Dropout(0.5))                                  # Applies dropout regularization to prevent overfitting
tuned_ai_model.add(tf.keras.layers.Dense(256, activation='relu'))
tuned_ai_model.add(tf.keras.layers.Dropout(0.5))

tuned_ai_model.add(tf.keras.layers.Dense(3, activation='softmax'))                # Adds the final fully connected layer with 3 units and softmax activation

tuned_ai_model.compile(loss='SparseCategoricalCrossentropy',                      # Specifies the loss function for trainin
              optimizer=tf.keras.optimizers.Adam(0.001),                       # Sets the optimizer (Adam) and learning rate
              metrics=['acc'])                                                 # Specifies the metrics to evaluate the model's performance
tuned_ai_model.summary()
Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 vgg16 (Functional)          (None, 7, 7, 512)         14714688  
                                                                 
 global_average_pooling2d (  (None, 512)               0         
 GlobalAveragePooling2D)                                         
                                                                 
 flatten (Flatten)           (None, 512)               0         
                                                                 
 dense (Dense)               (None, 256)               131328    
                                                                 
 dropout (Dropout)           (None, 256)               0         
                                                                 
 dense_1 (Dense)             (None, 256)               65792     
                                                                 
 dropout_1 (Dropout)         (None, 256)               0         
                                                                 
 dense_2 (Dense)             (None, 3)                 771       
                                                                 
=================================================================
Total params: 14912579 (56.89 MB)
Trainable params: 197891 (773.01 KB)
Non-trainable params: 14714688 (56.13 MB)
_________________________________________________________________
In [ ]:
seed_value = 42
random.seed(seed_value)
np.random.seed(seed_value)
tf.random.set_seed(seed_value)

# defines file path to save the best model weights, define accuracy metric, verbose display updates of the model and saves best model
checkpoint = tf.keras.callbacks.ModelCheckpoint('model/tuned_ai_model_best.saved', monitor='acc', verbose=1, mode='max',save_best_only=True)
# stops the model training if there is no improvement in metrics
early = tf.keras.callbacks.EarlyStopping(monitor="val_loss", mode="min",restore_best_weights=True, patience=10)

callbacks_list = [checkpoint,early]

# training the model
history = tuned_ai_model.fit(                                                     # fit the model on training data
        aug_train_data,
        validation_data = test_data,
        epochs=50,                                                             # no of epochs to train the model
        shuffle=False,
        verbose=True,                                                          # Prints updates during training.
        callbacks=callbacks_list
)
Epoch 1/50
4/4 [==============================] - ETA: 0s - loss: 1.2529 - acc: 0.3896
Epoch 1: acc improved from -inf to 0.38961, saving model to model/tuned_ai_model_best.saved
4/4 [==============================] - 14s 3s/step - loss: 1.2529 - acc: 0.3896 - val_loss: 1.1280 - val_acc: 0.3939
Epoch 2/50
4/4 [==============================] - ETA: 0s - loss: 1.1196 - acc: 0.4416
Epoch 2: acc improved from 0.38961 to 0.44156, saving model to model/tuned_ai_model_best.saved
4/4 [==============================] - 11s 3s/step - loss: 1.1196 - acc: 0.4416 - val_loss: 0.9905 - val_acc: 0.5000
Epoch 3/50
4/4 [==============================] - ETA: 0s - loss: 1.1166 - acc: 0.4242
Epoch 3: acc did not improve from 0.44156
4/4 [==============================] - 8s 2s/step - loss: 1.1166 - acc: 0.4242 - val_loss: 0.9414 - val_acc: 0.3939
Epoch 4/50
4/4 [==============================] - ETA: 0s - loss: 0.9544 - acc: 0.5455
Epoch 4: acc improved from 0.44156 to 0.54545, saving model to model/tuned_ai_model_best.saved
4/4 [==============================] - 11s 3s/step - loss: 0.9544 - acc: 0.5455 - val_loss: 0.8936 - val_acc: 0.5606
Epoch 5/50
4/4 [==============================] - ETA: 0s - loss: 0.8810 - acc: 0.5714
Epoch 5: acc improved from 0.54545 to 0.57143, saving model to model/tuned_ai_model_best.saved
4/4 [==============================] - 11s 3s/step - loss: 0.8810 - acc: 0.5714 - val_loss: 0.8254 - val_acc: 0.7879
Epoch 6/50
4/4 [==============================] - ETA: 0s - loss: 0.8534 - acc: 0.6017
Epoch 6: acc improved from 0.57143 to 0.60173, saving model to model/tuned_ai_model_best.saved
4/4 [==============================] - 10s 3s/step - loss: 0.8534 - acc: 0.6017 - val_loss: 0.7559 - val_acc: 0.7727
Epoch 7/50
4/4 [==============================] - ETA: 0s - loss: 0.8136 - acc: 0.6061
Epoch 7: acc improved from 0.60173 to 0.60606, saving model to model/tuned_ai_model_best.saved
4/4 [==============================] - 11s 3s/step - loss: 0.8136 - acc: 0.6061 - val_loss: 0.7016 - val_acc: 0.7121
Epoch 8/50
4/4 [==============================] - ETA: 0s - loss: 0.7333 - acc: 0.7056
Epoch 8: acc improved from 0.60606 to 0.70563, saving model to model/tuned_ai_model_best.saved
4/4 [==============================] - 12s 3s/step - loss: 0.7333 - acc: 0.7056 - val_loss: 0.6376 - val_acc: 0.7424
Epoch 9/50
4/4 [==============================] - ETA: 0s - loss: 0.7231 - acc: 0.6926
Epoch 9: acc did not improve from 0.70563
4/4 [==============================] - 8s 2s/step - loss: 0.7231 - acc: 0.6926 - val_loss: 0.5969 - val_acc: 0.8030
Epoch 10/50
4/4 [==============================] - ETA: 0s - loss: 0.6417 - acc: 0.7273
Epoch 10: acc improved from 0.70563 to 0.72727, saving model to model/tuned_ai_model_best.saved
4/4 [==============================] - 10s 3s/step - loss: 0.6417 - acc: 0.7273 - val_loss: 0.5676 - val_acc: 0.8788
Epoch 11/50
4/4 [==============================] - ETA: 0s - loss: 0.6532 - acc: 0.7143
Epoch 11: acc did not improve from 0.72727
4/4 [==============================] - 9s 2s/step - loss: 0.6532 - acc: 0.7143 - val_loss: 0.5287 - val_acc: 0.8030
Epoch 12/50
4/4 [==============================] - ETA: 0s - loss: 0.6107 - acc: 0.7532
Epoch 12: acc improved from 0.72727 to 0.75325, saving model to model/tuned_ai_model_best.saved
4/4 [==============================] - 11s 3s/step - loss: 0.6107 - acc: 0.7532 - val_loss: 0.5269 - val_acc: 0.7727
Epoch 13/50
4/4 [==============================] - ETA: 0s - loss: 0.5414 - acc: 0.7403
Epoch 13: acc did not improve from 0.75325
4/4 [==============================] - 8s 2s/step - loss: 0.5414 - acc: 0.7403 - val_loss: 0.4948 - val_acc: 0.7727
Epoch 14/50
4/4 [==============================] - ETA: 0s - loss: 0.6182 - acc: 0.7489
Epoch 14: acc did not improve from 0.75325
4/4 [==============================] - 9s 2s/step - loss: 0.6182 - acc: 0.7489 - val_loss: 0.4592 - val_acc: 0.8788
Epoch 15/50
4/4 [==============================] - ETA: 0s - loss: 0.5038 - acc: 0.8225
Epoch 15: acc improved from 0.75325 to 0.82251, saving model to model/tuned_ai_model_best.saved
4/4 [==============================] - 10s 3s/step - loss: 0.5038 - acc: 0.8225 - val_loss: 0.4482 - val_acc: 0.8333
Epoch 16/50
4/4 [==============================] - ETA: 0s - loss: 0.5679 - acc: 0.7749
Epoch 16: acc did not improve from 0.82251
4/4 [==============================] - 9s 2s/step - loss: 0.5679 - acc: 0.7749 - val_loss: 0.4419 - val_acc: 0.7879
Epoch 17/50
4/4 [==============================] - ETA: 0s - loss: 0.4788 - acc: 0.8268
Epoch 17: acc improved from 0.82251 to 0.82684, saving model to model/tuned_ai_model_best.saved
4/4 [==============================] - 11s 3s/step - loss: 0.4788 - acc: 0.8268 - val_loss: 0.4309 - val_acc: 0.8182
Epoch 18/50
4/4 [==============================] - ETA: 0s - loss: 0.4669 - acc: 0.8095
Epoch 18: acc did not improve from 0.82684
4/4 [==============================] - 8s 2s/step - loss: 0.4669 - acc: 0.8095 - val_loss: 0.4124 - val_acc: 0.8485
Epoch 19/50
4/4 [==============================] - ETA: 0s - loss: 0.4539 - acc: 0.8355
Epoch 19: acc improved from 0.82684 to 0.83550, saving model to model/tuned_ai_model_best.saved
4/4 [==============================] - 11s 3s/step - loss: 0.4539 - acc: 0.8355 - val_loss: 0.3955 - val_acc: 0.8485
Epoch 20/50
4/4 [==============================] - ETA: 0s - loss: 0.4495 - acc: 0.8442
Epoch 20: acc improved from 0.83550 to 0.84416, saving model to model/tuned_ai_model_best.saved
4/4 [==============================] - 11s 3s/step - loss: 0.4495 - acc: 0.8442 - val_loss: 0.3390 - val_acc: 0.9091
Epoch 21/50
4/4 [==============================] - ETA: 0s - loss: 0.4438 - acc: 0.7965
Epoch 21: acc did not improve from 0.84416
4/4 [==============================] - 10s 2s/step - loss: 0.4438 - acc: 0.7965 - val_loss: 0.4142 - val_acc: 0.8182
Epoch 22/50
4/4 [==============================] - ETA: 0s - loss: 0.4630 - acc: 0.8268
Epoch 22: acc did not improve from 0.84416
4/4 [==============================] - 9s 2s/step - loss: 0.4630 - acc: 0.8268 - val_loss: 0.3786 - val_acc: 0.8636
Epoch 23/50
4/4 [==============================] - ETA: 0s - loss: 0.4295 - acc: 0.8398
Epoch 23: acc did not improve from 0.84416
4/4 [==============================] - 8s 2s/step - loss: 0.4295 - acc: 0.8398 - val_loss: 0.3200 - val_acc: 0.9242
Epoch 24/50
4/4 [==============================] - ETA: 0s - loss: 0.4406 - acc: 0.8312
Epoch 24: acc did not improve from 0.84416
4/4 [==============================] - 9s 2s/step - loss: 0.4406 - acc: 0.8312 - val_loss: 0.3242 - val_acc: 0.8788
Epoch 25/50
4/4 [==============================] - ETA: 0s - loss: 0.4235 - acc: 0.8442
Epoch 25: acc did not improve from 0.84416
4/4 [==============================] - 8s 2s/step - loss: 0.4235 - acc: 0.8442 - val_loss: 0.3058 - val_acc: 0.9091
Epoch 26/50
4/4 [==============================] - ETA: 0s - loss: 0.3849 - acc: 0.8485
Epoch 26: acc improved from 0.84416 to 0.84848, saving model to model/tuned_ai_model_best.saved
4/4 [==============================] - 12s 3s/step - loss: 0.3849 - acc: 0.8485 - val_loss: 0.3554 - val_acc: 0.8485
Epoch 27/50
4/4 [==============================] - ETA: 0s - loss: 0.3862 - acc: 0.8658
Epoch 27: acc improved from 0.84848 to 0.86580, saving model to model/tuned_ai_model_best.saved
4/4 [==============================] - 10s 3s/step - loss: 0.3862 - acc: 0.8658 - val_loss: 0.3457 - val_acc: 0.8636
Epoch 28/50
4/4 [==============================] - ETA: 0s - loss: 0.4212 - acc: 0.8268
Epoch 28: acc did not improve from 0.86580
4/4 [==============================] - 8s 2s/step - loss: 0.4212 - acc: 0.8268 - val_loss: 0.3169 - val_acc: 0.8636
Epoch 29/50
4/4 [==============================] - ETA: 0s - loss: 0.3753 - acc: 0.8615
Epoch 29: acc did not improve from 0.86580
4/4 [==============================] - 10s 2s/step - loss: 0.3753 - acc: 0.8615 - val_loss: 0.2831 - val_acc: 0.9242
Epoch 30/50
4/4 [==============================] - ETA: 0s - loss: 0.3731 - acc: 0.8615
Epoch 30: acc did not improve from 0.86580
4/4 [==============================] - 8s 2s/step - loss: 0.3731 - acc: 0.8615 - val_loss: 0.2829 - val_acc: 0.9242
Epoch 31/50
4/4 [==============================] - ETA: 0s - loss: 0.3541 - acc: 0.8571
Epoch 31: acc did not improve from 0.86580
4/4 [==============================] - 9s 2s/step - loss: 0.3541 - acc: 0.8571 - val_loss: 0.3898 - val_acc: 0.8485
Epoch 32/50
4/4 [==============================] - ETA: 0s - loss: 0.3886 - acc: 0.8701
Epoch 32: acc improved from 0.86580 to 0.87013, saving model to model/tuned_ai_model_best.saved
4/4 [==============================] - 11s 3s/step - loss: 0.3886 - acc: 0.8701 - val_loss: 0.2987 - val_acc: 0.8636
Epoch 33/50
4/4 [==============================] - ETA: 0s - loss: 0.3290 - acc: 0.8615
Epoch 33: acc did not improve from 0.87013
4/4 [==============================] - 8s 2s/step - loss: 0.3290 - acc: 0.8615 - val_loss: 0.2649 - val_acc: 0.9242
Epoch 34/50
4/4 [==============================] - ETA: 0s - loss: 0.3790 - acc: 0.8528
Epoch 34: acc did not improve from 0.87013
4/4 [==============================] - 7s 2s/step - loss: 0.3790 - acc: 0.8528 - val_loss: 0.2823 - val_acc: 0.9091
Epoch 35/50
4/4 [==============================] - ETA: 0s - loss: 0.3554 - acc: 0.8701
Epoch 35: acc did not improve from 0.87013
4/4 [==============================] - 9s 2s/step - loss: 0.3554 - acc: 0.8701 - val_loss: 0.2919 - val_acc: 0.8636
Epoch 36/50
4/4 [==============================] - ETA: 0s - loss: 0.3349 - acc: 0.8701
Epoch 36: acc did not improve from 0.87013
4/4 [==============================] - 7s 2s/step - loss: 0.3349 - acc: 0.8701 - val_loss: 0.2772 - val_acc: 0.8788
Epoch 37/50
4/4 [==============================] - ETA: 0s - loss: 0.3280 - acc: 0.8745
Epoch 37: acc improved from 0.87013 to 0.87446, saving model to model/tuned_ai_model_best.saved
4/4 [==============================] - 12s 3s/step - loss: 0.3280 - acc: 0.8745 - val_loss: 0.2984 - val_acc: 0.8636
Epoch 38/50
4/4 [==============================] - ETA: 0s - loss: 0.3329 - acc: 0.8571
Epoch 38: acc did not improve from 0.87446
4/4 [==============================] - 7s 2s/step - loss: 0.3329 - acc: 0.8571 - val_loss: 0.2996 - val_acc: 0.8636
Epoch 39/50
4/4 [==============================] - ETA: 0s - loss: 0.3068 - acc: 0.8874
Epoch 39: acc improved from 0.87446 to 0.88745, saving model to model/tuned_ai_model_best.saved
4/4 [==============================] - 11s 3s/step - loss: 0.3068 - acc: 0.8874 - val_loss: 0.2438 - val_acc: 0.9394
Epoch 40/50
4/4 [==============================] - ETA: 0s - loss: 0.2756 - acc: 0.9091
Epoch 40: acc improved from 0.88745 to 0.90909, saving model to model/tuned_ai_model_best.saved
4/4 [==============================] - 11s 3s/step - loss: 0.2756 - acc: 0.9091 - val_loss: 0.2760 - val_acc: 0.8788
Epoch 41/50
4/4 [==============================] - ETA: 0s - loss: 0.2485 - acc: 0.9134
Epoch 41: acc improved from 0.90909 to 0.91342, saving model to model/tuned_ai_model_best.saved
4/4 [==============================] - 11s 3s/step - loss: 0.2485 - acc: 0.9134 - val_loss: 0.2650 - val_acc: 0.8939
Epoch 42/50
4/4 [==============================] - ETA: 0s - loss: 0.2913 - acc: 0.8874
Epoch 42: acc did not improve from 0.91342
4/4 [==============================] - 9s 2s/step - loss: 0.2913 - acc: 0.8874 - val_loss: 0.2480 - val_acc: 0.9091
Epoch 43/50
4/4 [==============================] - ETA: 0s - loss: 0.2733 - acc: 0.9048
Epoch 43: acc did not improve from 0.91342
4/4 [==============================] - 8s 2s/step - loss: 0.2733 - acc: 0.9048 - val_loss: 0.2343 - val_acc: 0.9242
Epoch 44/50
4/4 [==============================] - ETA: 0s - loss: 0.2729 - acc: 0.8961
Epoch 44: acc did not improve from 0.91342
4/4 [==============================] - 7s 2s/step - loss: 0.2729 - acc: 0.8961 - val_loss: 0.2316 - val_acc: 0.9242
Epoch 45/50
4/4 [==============================] - ETA: 0s - loss: 0.2519 - acc: 0.9004
Epoch 45: acc did not improve from 0.91342
4/4 [==============================] - 10s 2s/step - loss: 0.2519 - acc: 0.9004 - val_loss: 0.2498 - val_acc: 0.8788
Epoch 46/50
4/4 [==============================] - ETA: 0s - loss: 0.2413 - acc: 0.9221
Epoch 46: acc improved from 0.91342 to 0.92208, saving model to model/tuned_ai_model_best.saved
4/4 [==============================] - 11s 3s/step - loss: 0.2413 - acc: 0.9221 - val_loss: 0.3430 - val_acc: 0.8485
Epoch 47/50
4/4 [==============================] - ETA: 0s - loss: 0.2682 - acc: 0.9004
Epoch 47: acc did not improve from 0.92208
4/4 [==============================] - 8s 2s/step - loss: 0.2682 - acc: 0.9004 - val_loss: 0.2117 - val_acc: 0.9242
Epoch 48/50
4/4 [==============================] - ETA: 0s - loss: 0.2707 - acc: 0.8961
Epoch 48: acc did not improve from 0.92208
4/4 [==============================] - 9s 2s/step - loss: 0.2707 - acc: 0.8961 - val_loss: 0.2072 - val_acc: 0.9242
Epoch 49/50
4/4 [==============================] - ETA: 0s - loss: 0.2772 - acc: 0.9221
Epoch 49: acc did not improve from 0.92208
4/4 [==============================] - 8s 2s/step - loss: 0.2772 - acc: 0.9221 - val_loss: 0.3180 - val_acc: 0.8636
Epoch 50/50
4/4 [==============================] - ETA: 0s - loss: 0.2423 - acc: 0.9177
Epoch 50: acc did not improve from 0.92208
4/4 [==============================] - 9s 2s/step - loss: 0.2423 - acc: 0.9177 - val_loss: 0.2397 - val_acc: 0.9091

Model Evaluation

In [ ]:
ytrain = np.array([])                                                          # Initialize an empty array for storing true labels of test data
xtrain = []                                                                    # Initialize an empty list for storing test data

for i in range(math.ceil(len(aug_train_data.classes)/batch_size)):             # Loop over test generator batches to extract test data and true labels
    xtrain.append(aug_train_data[i][0])                                        # Append test data to xtest list
    ytrain= np.concatenate((ytrain,aug_train_data[i][-1]))                     # Concatenate true labels to ytest array

xtrain = np.concatenate((xtrain),axis=0)                                       # Concatenate test data along the batch axis

ypred_prob_train_tune = tuned_ai_model.predict(xtrain)                                        # Predict probabilities for the test data
ypred_train_tune = np.argmax(ypred_prob_train_tune,axis=1)                                           # Predicted labels by selecting the class with the highest probability
8/8 [==============================] - 1s 119ms/step
In [ ]:
model_train_score = recall_score(ytrain, ypred_train_tune,average='macro')
print("Model Score on Train Data:", np.round(100*model_train_score, 2))
Model Score on Train Data: 91.07
In [ ]:
# Set the size of the figure for the heatmap
plt.figure(figsize=(6, 6))
# Compute the confusion matrix based on true and predicted labels
hm = sns.heatmap(confusion_matrix(ytrain,ypred_train_tune), annot=True, vmin=0, fmt='g', cmap='Blues', cbar=False,xticklabels=['Normal','Viral Penumonia','Covid'],yticklabels=['Normal','Viral Penumonia','Covid'])
hm.set(xlabel='Predicted labels') # Set the x-axis label for the heatmap
hm.set(ylabel='True label') # # Set the y-axis label for the heatmap
Out[ ]:
[Text(45.72222222222221, 0.5, 'True label')]
In [ ]:
ytest = np.array([])                                                           # Initialize an empty array for storing true labels of test data
xtest = []                                                                     # Initialize an empty list for storing test data

for i in range(math.ceil(len(test_data.classes)/batch_size)):                  # Loop over test generator batches to extract test data and true labels
    xtest.append(test_data[i][0])                                              # Append test data to xtest list
    ytest= np.concatenate((ytest,test_data[i][-1]))                            # Concatenate true labels to ytest array

xtest = np.concatenate((xtest),axis=0)                                         # Concatenate test data along the batch axis

ypred_prob_test_tune =tuned_ai_model.predict(xtest)                                    # Predict probabilities for the test data
ypred_test_tune = np.argmax(ypred_prob_test_tune,axis=1)                                           # Predicted labels by selecting the class with the highest probability
3/3 [==============================] - 0s 133ms/step
In [ ]:
model_test_score = recall_score(ytest, ypred_test_tune,average='macro')
print("Model Score on Test Data:", np.round(100*model_test_score, 2))
Model Score on Test Data: 90.0
In [ ]:
# Set the size of the figure for the heatmap
plt.figure(figsize=(6, 6))
# Compute the confusion matrix based on true and predicted labels
hm = sns.heatmap(confusion_matrix(ytest, ypred_test_tune), annot=True, vmin=0, fmt='g', cmap='Blues', cbar=False,xticklabels=['Normal','Viral Penumonia','Covid'],yticklabels=['Normal','Viral Penumonia','Covid'])
hm.set(xlabel='Predicted labels') # Set the x-axis label for the heatmap
hm.set(ylabel='True label') # # Set the y-axis label for the heatmap
Out[ ]:
[Text(45.72222222222221, 0.5, 'True label')]

Model Testing¶

Model testing is important for:

  • Validating model performance
  • Identifying and mitigating errors or flaws
  • Assessing model robustness and generalizability
  • Building user trust and confidence

Model Testing¶

In [ ]:
ytest = np.array([])
xtest = []

for i in range(math.ceil(len(test_data.classes)/batch_size)):
    xtest.append(test_data[i][0])
    ytest= np.concatenate((ytest,test_data[i][-1]))

xtest = np.concatenate((xtest),axis=0)

ypred_prob = tuned_ai_model.predict(xtest)
ypred = np.argmax(ypred_prob,axis=1)

model_test_score = recall_score(ytest, ypred_test_tune,average='macro')
print("Model Score on Test Data:", np.round(100*model_test_score, 2), '\n\n')

plt.figure(figsize=(6, 6))
hm = sns.heatmap(confusion_matrix(ytest,ypred), annot=True, vmin=0, fmt='g', cmap='Blues', cbar=False,
            xticklabels=['Normal','Viral Penumonia','Covid'],yticklabels=['Normal','Viral Penumonia','Covid'])
hm.set(xlabel='Predicted_labels')
hm.set(ylabel='True_labels');
3/3 [==============================] - 0s 108ms/step
Model Score on Test Data: 90.0 


Observations from Model Testing¶

  • Our AI model has a score of ~90% on the test data, which is a good score.
  • We can now move ahead to the next stage.

Model Deployment and Monitoring¶

s50.PNG

Model deployment helps with the following:

  • Realizing Value
  • Scalability and Efficiency
  • Continuous Learning and Improvement
  • Business Impact and Decision Support

Model Deployment¶

In [ ]:
# path to the save the trained model
saved_model_path = "/content/drive/covid_detector.joblib"

# saving the final model
joblib.dump(tuned_ai_model, saved_model_path)
Out[ ]:
['/content/drive/MyDrive/Colab Notebooks/AI Application Case Study - Image Data/covid_detector.joblib']
In [ ]:
# path to the save the trained model
saved_model_path = "/content/drive/covid_detector.joblib"

# loading the saved model
covid_detector = joblib.load(saved_model_path)
In [ ]:
# creating the deployment input interface
image = gr.Image()

# creating the deployment output interface
label = gr.Label(num_top_classes=3)
In [ ]:
# defining the dimensions of the input image
height = 224
width = 224
dimensions = (width, height)

# defining the class names for predictions
class_names = {0: 'Normal', 1: 'Viral Pneumonia', 2: 'Covid'}

# define a function that will take the necessary inputs and make predictions
def predict_covid(image):
    # resizing the input image
    image = cv2.resize(image, dimensions, interpolation=cv2.INTER_LINEAR)
    image = image / 255.0
    # reshaping the image to match the model's input shape
    image = image.reshape((-1, 224, 224, 3))
    # making predictions using the loaded model
    prediction = covid_detector.predict(image).flatten()
    # formatting the results to return final results as class names
    return {class_names[i]: float(prediction[i]) for i in range(3)}
In [ ]:
# defining the structure of the deployment interface and how the components will interact
demo = gr.Interface(
    fn=predict_covid,
    inputs = image,
    outputs = label,
    title="COVID Detection",
    description= "This interface will predict whether a given patient is normal, has viral pneumonia, or has COVID based on chest X-ray scan provided.",
    allow_flagging="never"
)
In [ ]:
# deploying the model
demo.launch(inline=False, share=True, debug=True)
In [ ]:
# shutting down the deployed model
demo.close()

Types of Model Deployment¶

There are generally two main modes of making predictions with a deployed AI model:

  • Batch Prediction: In batch prediction mode, predictions are made on a batch of input data all at once. This mode is suitable when you have a large set of data that needs predictions in a batch process, such as running predictions on historical data or performing bulk predictions on a scheduled basis.

  • Real-time (or Interactive) Prediction: In real-time or interactive prediction mode, predictions are made on individual data points in real-time as they arrive. This mode is suitable when you need immediate or on-demand predictions for new and incoming data.

The choice of prediction mode depends on the specific requirements and use case of the deployed AI model. Batch prediction is preferable when efficiency in processing large volumes of data is important, while real-time prediction is suitable for scenarios that require immediate or interactive responses to new data.

Metrics and Dashboarding¶

metrics.png

Metrics and dashboarding are the tools that businesses use to track their performance. Some of the benefits of using metrics and dashboarding:

  • Improved decision-making
  • Increased efficiency
  • Increased visibility

Dashboard for the Data Team¶

image.png

Note: The above chart is indicative in nature.

Decision Making¶

Decision_makinf.png

We have built an AI model, tested it, deployed it, and used the model's outputs to visualize the important business metrics via dashboards. Now the final step is to use the AI model for decision-making and determine the impact of implementing the AI solution.

The trends of model performance useful for the Data Team. They can use it to

  • monitor the model's performance over time
  • set thresholds for the acceptable lower limit of model performance
  • decide when to retrain the model

Healthcare personnel (doctors, radiologists, etc.) can use the AI solution to take a call on cases where the model's predictions are on the edge.

  • For example, based on the X-ray scan, the model predicts that there is a 40% chance that the person has COVID, the heathcare personnel can take a final call by examining the X-ray scan and model output together
  • This can ensure that the COVID detection is more accurate

Let's assess the impact of our AI solution.

  • The model can make a prediction regarding the presence of COVID based on an X-ray image in less than 5 seconds
  • The AI solution can yield results within 15 minutes of the X-ray scan, compared to the 4-hour TAT of an RT-PCR test
  • This shows that the AI solution can provide a preliminary COVID diagnosis 16 times faster than traditional methods.
  • Assuming that 50 patients get screened using RT-PCR every hour and 10 samples can be processed at once, it would take approx. two days to get the test results of the 50 patients.
    • We have considered 12 working hours a day
  • For the same 50 patients, assuming that we can process only one X-ray scan at a time, it would take approx. a day to get the test results of the 50 patients.
  • So, we can get the preliminary diagnosis of patients done in half the time with the AI solution.

Note: The above numbers are indicative in nature.

**Happy Learning!**